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ABSTRACT 

Accounting educators continuously seek ways to effectively integrate instructional technology 
into accounting coursework as a means to facilitate active learning environments and address the 
technology-driven learning preferences of the current generation of students. Most accounting 
textbook publishers now provide interactive, web-based learning systems as a supplement to their 
textbooks. However, there are critical differences across publishers with respect to the system’s 
interface, functions, content, features, and support. These differences directly impact the effectiveness 
of a web-based learning system as an instructional tool and as a corollary, may impact the ultimate 
utility of the associated textbook as a pedagogical resource. As such, an evaluation of available web- 
based learning systems is an essential component of the textbook review process and should consist of 
a meticulous evaluation of the system’s functionality and features in light of instructor and student 
needs and preferences. Unfortunately, given the functional intricacies and disparity in features across 
web-based learning systems, this can be a daunting task. The purpose of this paper is to offer 
accounting educators some practical guidance on evaluating web-based learning systems prior to 
adoption. This guidance is presented in the form of a framework based on key processes underlying the 
systems development life cycle (SDLC), a logical process model used by systems professionals to 
guide the acquisition and/or development of information systems. 
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INTRODUCTION 

Meeting the educational needs of students is a constant pedagogical challenge, particularly in 
courses that are rule-based and complex such as accounting (Lippincott, Matulich, & Squires 2007). 
Educators must continuously adapt to student learning styles and preferences to effectively engage 
them in the learning process. This requires educators to first, be aware of student learning preferences 
and then, introduce innovative ways to meet those preferences (Matulich, Papp, & Haytko 2008). The 
current generation of students, commonly referred to as “NetGen” learners, grew up surrounded by 
technology and bombarded by continuous technological advancements. Computers, multimedia, the 
Internet, cell phones, and computer games were and continue to be a central and intuitive part of their 
life. NetGen learners stay connected and their experiences are interactive and real time; as a result, 
they have little tolerance for delays, non-interactive environments, or lack of current technology 
(Oblinger 2003, Papp, 2010). They crave stimulation, support, and immediate feedback and have 
developed a trial-and-error style of experiential learning that has its roots in computer gaming wherein 
the fastest way to learn (i.e., master the game) is to fail and try again (Arhin & Johnson-Mallard 2003, 
Oblinger 2003, Matulich, Papp, & Haytko 2008). They also prefer self-paced, any-time-any-place 
learning environments in lieu of traditional structured classroom pedagogy (Matulich, Papp, & Haytko 
2008). These factors have resulted in a student group that responds best to interactive learning 
environments in which they receive real-time feedback and can control the pace and depth of their 
learning; moreover, the current generation of learners expect to use technology to facilitate learning 
(Arhin & Johnson-Mallard 2003, Lippincott, Matulich, & Squires 2007, Lippincott, Pergola, & 

Squires 2006, Matulich, Papp, & Haytko 2008, Papp, 2010). As such, it should come as no great 
surprise that traditional textbook-based accounting pedagogy which relies on textbook readings, one¬ 
way lectures, and passive in-class problem-solving are less effective with this current generation of 
accounting students (Eisner 2004, Bryant & Hunton 2000, Lippincott, Matulich, & Squires 2007, 
Matulich, Papp, & Haytko 2008). 

The integration of interactive instructional technology into accounting coursework as a 
supplement to traditional textbook pedagogy provides an effective means to address the technology- 
driven, interactive learning preferences of the current generation of accounting students (Lippincott, 
Matulich, & Squires, 2007, Lippincott, Pergola, & Squires 2006, Matulich, Papp, & Haytko 2008). 

The use of interactive instructional technology as a learning tool is strongly supported by behavioral 
and cognitive theory which suggests that reinforcement, active participation, and learner control are 
critical components of the learning process (Bryant & Hunton 2000, Thompson , Simonson, & 
Hargrave 1992). Behavioral theory holds that positive or negative reinforcement immediately 
following stimulus prompts learning suggesting that an instructional technology that provides real-time 
corrective feedback during problem solving will enhance learning for accounting students (Bryant & 
Hunton 2000). Cognitive theory holds that interactive participation heightens learning suggesting that 
an instructional technology that actively engages accounting students by way of two-way exchanges 
and an appropriate level of learner control will have a positive impact on learning (Bryant & Hunton 
2000). Instructional systems that allow for such interactivity, learner control, and real-time feedback 
are now widely available in the form of web-based learning systems offered as supplements to 
accounting textbooks (e.g., Cengage: ThomsonNow,CengageNOW; Pearson: MyAccountingLab; 
Wiley: WileyPLUS; McGraw-Hill: Homework Manager/Connect). 

Most accounting textbook publishers now offer interactive, web-based learning systems as a 
supplement to their accounting textbooks (e.g., see http://www.cengage.com/highered/ , 
http://www.pearsonhighered.com/educator , http://www.wiley.com/WileyCDA/ , and 
http://connect.mcgraw-hill.com/ ). These web-based learning systems typically offer online learning 
environments that may include interactive book content, study management tools, multimedia 
activities, homework assignments, self-assessment tools, and grade book options. However, there are 
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critical differences across publishers with respect to system functions, content, capabilities, and 
support. These features can directly impact the effectiveness of a web-based learning system as an 
instructional tool for facilitating interactive learning and moreover, may impact the ultimate utility of 
the associated textbook as a pedagogical resource (Lippincott, Pergola, & Squires 2006, Pergola & 
Squires 2007). As such, an evaluation of available web-based learning systems is an essential 
component of the textbook review and adoption process and should consist of a meticulous evaluation 
of the system in light of both instructor and student preferences. Unfortunately, given the functional 
intricacies and disparity in content and features across available web-based learning systems, this can 
be an overwhelming task. 

The purpose of this paper is to offer accounting educators some practical guidance on 
evaluating web-based learning systems prior to adoption. This guidance is presented in the form of a 
framework based on key processes underlying the systems development life cycle (SDLC), a logical 
process model used by systems professionals to guide the acquisition and/or development of 
information systems. The remainder of this paper is structured as follows. The next section discusses 
the relevance of the SDLC for evaluating web-based learning systems offered with accounting 
textbooks. The following sections apply key SDLC processes to the evaluation of web-based learning 
systems and discuss an assessment framework developed to assist accounting educators in evaluating 
such systems. The final section of the paper presents brief concluding remarks followed by an 
appendix containing the assessment framework. 

THE SDLC AND EVALUATION OF WEB-BASED LEARNING SYSTEMS 

The SDLC is a logical process model describing the conceptual phases of an information 
systems development endeavor from inception to completion (see Dennis, Wixom, & Roth 2009, 
Gelinas & Dull 2010, Hall 2011, Shelly & Rosenblatt 2010). The SDLC is used by systems 
professionals as an abstract framework to manage systems development projects to ensure that 
development processes are efficacious and that resulting systems meet organizational needs (Gelinas & 
Dull 2010). There is some variability in the definition, perceived importance, and application of SDLC 
phases, especially across differing development methods (e.g., sequential vs. iterative vs. agile 
methodologies); however, the SDLC is generally considered to include systems survey, systems 
analysis, systems design, systems implementation, and systems operations phases (see Dennis, Wixom, 
& Roth 2009, Fitzgerald, Russo, & Stolterman 2002, Gelinas & Dull 2010, Shelly & Rosenblatt 2010). 

SDLC concepts are applicable to the in-house development of information systems as well as 
the acquisition of information systems solutions from external parties (Gelinas & Dull 2010). The 
SDLC phases applicable to a system acquisition project are analogous to system development phases; 
however, since the focus is on acquiring (rather than developing) systems solutions, certain 
subprocesses within the design phase are adapted to focus on acquisition concerns relating to the 
identification and evaluation of feasible solutions and the selection of a solution that best meets system 
requirements (Gelinas, Sutton, & Hunton, 2005). Since, the adoption of a web-based learning system is 
essentially a systems acquisition decision, SDLC concepts as adapted for systems acquisition 
considerations may be used to guide the process of evaluating available solutions and determining a 
final solution that best meets the pedagogical objectives and requirements of accounting instructors 
and students. The SDLC phases most relevant to evaluating web-based learning systems include 
systems survey, system analysis, and systems selection processes. Each is discussed with respect to 
evaluating web-based learning systems below. 


Evaluating web-based learning systems, Page 3 


Journal of Instructional Pedagogies 


SYSTEMS SURVEY 

The systems survey is a preliminary planning process that involves defining problems, 
establishing objectives, determining feasibility of proposed solutions, proposing a course of action, and 
devising a project plan (Dennis, Wixom, & Roth 2009, Gelinas & Dull 2010, Gelinas, Sutton, & 
Hunton, 2005, Shelly & Rosenblatt 2010). With respect to evaluating web-base learning systems, key 
aspects would involve defining pedagogical problem areas, determining pedagogical objectives, and 
assessing the feasibility of a web-based learning system as a solution. 

Problem Identification & Objectives 

As mentioned above, the central problem with traditional textbook-based accounting pedagogy 
is that it does not meet learning preferences of the current generation of accounting students. The 
traditional approach to teaching accounting involves assigned textbook readings, instructor lectures, 
and end-of-chapter homework assignments followed by instructor review of homework assignments 
during class. Lectures typically result in a one-way exchange of information as large class sizes along 
with time constraints often limit interactivity. Homework assignments typically involve multiple 
exercises and/or problems selected and assigned from each chapter and are typically completed by 
students in paper-and-pencil form. Instructors generally review homework solutions during normal 
class meeting times, but may or may not collect, grade, and provide individual feedback on student 
homework performance. If homework is not collected and graded, the primary feedback mechanism 
becomes exam performance which means that feedback occurs subsequent to the central learning 
process. Attempts to grade and review homework to allow for adequate feedback during learning 
results in instructor inefficiencies in the form of time spent grading and recording homework, class 
time spent reviewing homework, and sometimes significant delays in the feedback provided to 
students. In a nutshell, the traditional approach to accounting pedagogy is typically characterized by an 
absence of technology, insignificant interactivity, nominal learner control over the pace of learning, 
and delayed feedback which results in a pedagogical approach diametrically opposed to the learning 
preferences of current accounting students. Key problems may be summarized as follows (Matulich, 
Papp, & Haytko 2008, Lippincott, Matulich, & Squires 2007) 

• The traditional textbook medium may not motivate students as current students prefer use of 
online technology. 

• Traditional lectures are passive which may result in lack of attention and retention as current 
students prefer interactive two-way exchanges. 

• Lack of control over the pace of learning frustrates students as current students prefer self- 
paced learning. 

• Lack of immediate feedback may impede learning and reinforce incorrect solution approaches 
as current students prefer real-time feedback as support for computer-game like trial-and-error 
learning approaches. 

• Attempts to provide feedback to students result in instructor inefficiencies in the form of time 
spend grading/recording homework, class time spend reviewing homework, and delays 
providing individual feedback. 

In light of these problems, the general objectives of acquiring a web-based learning system 
would be to first, provide a learning approach that supports the technology, interactivity, learner 
control, and feedback preferences of current students and second, to provide a pedagogical tool to 
improve instructor efficacy in supporting such preferences. 
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Feasibility Assessment 

Feasibility assessment involves determining the practicability of addressing identified problems 
and objectives with the proposed solution. Three key aspects of feasibility are technical feasibility, 
operational feasibility, and economic feasibility (Gelinas & Dull 2010, Gelinas, Sutton, & Hunton, 
2005, Hall 2011). Economic feasibility refers to the costs of the proposed system, the availability of 
funds to implement the system, and how a system solution compares to other projects competing for 
available funds (Gelinas, Sutton, & Hunton, 2005, Hall 2011). Since web-based learning systems are 
supported on publisher websites, there is negligible cost to universities or instructors; however, 
students must purchase access to the web-based system so the central economic feasibility issue will be 
the cost of student access to web-based learning system resources. 

Technical feasibility relates to whether or not a problem has a technically feasible solution that 
can be addressed using available hardware and software technology (Gelinas, Sutton, & Hunton, 2005, 
Hall 2011). The web-based learning systems considered in this paper are developed by textbook 
publishers, supported on publisher platforms, and accessed via a web interface. High-speed Internet 
access and a compatible web browser are required (both of which are normally readily available within 
a university environment) but no additional investment in either hardware or software is necessary. 

Operational feasibility refers to consistency between the operational requirements of the new 
system and existing procedures and personnel (Gelinas, Sutton, & Hunton, 2005, Hall 2011). A new 
system may result in changes to procedures, need for new personnel, behavioral reactions to the 
change, and need for training. With respect to the web-based learning systems considered in this paper, 
relevant considerations include changes to instructional methods, instructor and student competency 
with technology, student and instructor reactions to the implementation of a web-based learning 
system, and university tolerance for alternative delivery models. A key operational feasibility concern 
is often change management as personnel will often resist a new system because they fear the 
implications of the change; as such, adequate training and user support are central to effectively 
managing technological change (O’Brien & Marakas 2010). With respect to web-based learning 
systems, some instructors may resist or even refuse to use a web-based learning system due to 
unfamiliarity with the system and/or impacts on their current pedagogical approach so system 
usability, training resources, and system support will be central considerations in evaluating the 
system. 

SYSTEM ANALYSIS: USER REQUIREMENTS 

System analysis involves specifying user requirements and defining system specifications to 
meet identified requirements (Dennis, Wixom, & Roth 2009, Gelinas & Dull 2010, Gelinas, Sutton, & 
Hunton, 2005, Shelly & Rosenblatt 2010).With respect to evaluating a web-base learning system, this 
would involve identifying student and instructor requirements for the system and translating user 
requirements into logical system specifications that may be used to guide the evaluation of different 
web-based learning solutions. The user requirements analysis should address the needs and 
preferences of all affected users (Gelinas & Dull 2010). In the case of a web-based learning system, 
this would include both instructors’ and students’ needs. Likely requirements were constructed based 
on basic functionality of web-based learning systems in light of the student learning preferences and 
problem definitions discussed above. These requirements are presented below from both student and 
instructor perspectives. 
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Student-Related Requirements 

The integration of web-based instructional technology into the curriculum should facilitate 
interactive learning, allow for student control and flexibility, and provide real-time feedback and 
guidance to meet the learning preferences of current accounting students. 

• The system should provide appropriate content, assignments, and study tools to help students 
manage learning. 

• The system should provide interactive access to textbook material relating to assignments or 
study tools. 

• The system should provide alternative delivery methods (different media, problem types, and 
study tools) to appeal to different learning styles. 

• The system should allow access to content, assignments, and study tools anytime from 
anywhere. 

• The system should allow user control over types of activities performed and pace at which 
activities are performed. 

• The system should provide immediate feedback to students as they work. 

• The system should provide problem solutions as study aids for students. 

• The system should provide self-assessment activities to help students determine progress. 

Instructor-Related Requirements 

A web-based instructional technology into the curriculum should also allow for instructor 
control and flexibility, promote interactive learning without direct intervention of the instructor, and 
improve instructor efficacy. 

• The system should provide an adequate selection of exercises, problems, and self-assessment 
exercises relating to chapter material. 

• The system should provide mix of different problem types to support learning objectives of the 
instructor. 

• The system should automatically grade student responses in the manner that instructors define. 

• The system should automatically provide correct answers and solution guidance to the student. 

• The system should support uploading and downloading of content to support instructor 
responses to student questions. 

• The system should allow instructors to review student performance and progress. 


SYSTEM ANALYSIS: SYSTEM SPECIFICATIONS 

There are innumerable considerations to take into account when performing a systems analysis 
of a web-based learning system, some of which have been identified in previous research (e.g., Pergola 
and Squires 2007). For the purposes of the framework developed in this paper, system specification 
considerations were derived from the feasibility concerns, student requirements, and instructor 
requirements discussed above with respect to basic functionality of web-based learning systems. 
General software selection considerations and previous research on electronic homework systems were 
also considered in identifying relevant considerations (ASA Research 2010, Pergola & Squires 2007). 
System specification considerations are organized into four categories: feasibility considerations, 
system functionality, available content, and relevant features and are briefly discussed below. 
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Feasibility Considerations 

Feasibility includes basic economic, technical, and operational feasibility concerns. Basic feasibility 
considerations for web-based learning systems includes cost, access code, and compatibility 
considerations. 

• Cost: The cost refers to the price of the access code. Prices vary significantly between 
products and some products have options pricing dependent on the content. Some vendors 
bundle the textbook with the access code, which can result in cost savings if properly 
negotiated. Most vendors allow direct purchase of the codes on their sites, which can 
potentially save students bookstore markups. 

• Access code life: Each system requires students to purchase an access code. As noted above, 
the code can be bundled with the book or students can buy the codes directly from the publisher 
web site. The life of the access code differs among vendors. Some allow multiple year access, 
some single-year, and some single-semester. The single semester code limit requires students 
to buy an additional code if they drop/fail the course and have to take it again in a subsequent 
semester. Inquiries about access code life and the vendor’s willingness to negotiate longer 
terms are important factors to consider. The price of the code and students’ economic status 
are additional issues that may impact how an instructor evaluates access code life. 

• Compatibility with other applications: As mentioned above, the web-based learning systems 
discussed in this paper are maintained on vendor platforms with a web interface and as such, 
are accessible via Internet access and a compatible web browser. However, it is important to 
ensure that they are compatible with university learning management programs (such as 
Blackboard or Web CT) and that they support exporting of grades and importing/exporting of 
content to other programs. 

Operational feasibility also includes training, product support, and vender reliability considerations. 

• Instructor training: The types of instructor training vary by product and include on-site 
training, web-based training, manuals, and peer support services. An advance copy of training 
materials or a request for a training demonstration can help instructors to evaluate the adequacy 
of the training. 

• Instructor support: There is a wide variation in the availability and type of technical support 
assistance. Types of support include interactive online support, email, and telephone support. 

• Hours of instructor support: Some products offer support from 9 to 5 Central Time, Monday 
through Friday. None of the products offer 24 hour 7 day a week live help. 

• Response rates: Statistics detailing response rates for each communication mode, including 
telephone wait time and email turnaround time, can help in the evaluation of response rates. 

• Instructor feedback: The vendors have different processes for instructor-reported errors, 
suggestions, and/or desired features. 

• Student enrollment and system use instructions: Students, especially freshman, have trouble 
enrolling if the process is unclear or complicated, resulting in instructor intervention, late 
assignments, and high levels of frustration. Preview of the student enrollment process and 
review of the student instructions can help assess the quality of the process. 

• Student technical support: There is a wide variation in the availability of technical support 
assistance. Some products offer support from 9 to 5 Central Time, Monday through Friday. 
None of the products offer 24 hour 7 day a week live help. 
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• Type of student support: Types of support include interactive online support, email, and 
telephone support. Student response rates for each communication mode, including telephone 
wait time and email turnaround statistics, are critical to the success of the system. Students 
may blame their instructor for system glitches which are beyond the instructor control. 

• Vendor Reliability: Students and instructors work from the publishers’ networks. Reliability 
is critical to students completing homework on time and easing frustration with the technology. 
In addition, if the network is overloaded because the publisher has oversold the capacity, 
students and instructors cannot access the site when needed. For these reasons, it is critically 
important to assess capacity and performance statistics. 

System Functionality 

Functional considerations include a number of factors relating to the ease of setup, system 
administration, and system use (Pergola & Squires, 2007). 

• Course administration procedures: On each system, courses must be created at the start of 
the semester and closed at the end of the semester. The processes differ among vendors. Some 
products define a school administrator who performs these functions; some products require the 
instructor to request course set up from the vendor. The latter process requires setup requests 2 
or 3 days in advance of when access the course is desired. It is important to ask about the setup 
process and response times if the vendor controls the process. 

• Ease of creating assignments: Each system contains certain end-of-chapter exercises and 
problems that can be used to create assignments electronically. Potential adopters of systems 
should review the process for creating assignments for ease of use. 

• Ease of authoring problems/exercises: Most systems allow instructors to add their own 
questions and problems. However, not all systems are user-friendly in this regard. The best 
way to assess this issue is to ask for a demonstration or for access to try it out. 

• Ability to support desired problem format: Most systems allow instructors to add their own 
problem sets but may have some restrictions with respect to problem format. Most support 
multiple choice formats and true/false but may not support more complex formatting. 

• Ability to change assignment parameters: Each system allows the user to define assignment 
parameters, such as the due date, the number of attempts allowed, whether the student can see 
the solution after the due date, whether the student can submit late assignments or not, etc. In 
some systems, changes to these parameters after students have begun the assignment results in 
lost grades. It is important to understand the system parameters, to determine which can be 
changed, when they can be changed, and the potential impact of those changes. 

• Student view: Some systems allow the instructor to toggle to “student view”. This allows the 
instructor to see what the students see. Some systems also treat the instructor as a student when 
in this view, which enables instructors to try problems in advance of assignments. This helps 
instructors determine if the course set up selection options created the desired result and to 
identify any grading problems. 

• Transfer of student access between classes: Students enroll in the electronic class related to 
their class section. Sometimes, students enroll in the wrong section or need to transfer their 
code from a class they dropped in the prior semester to the class in the current semester. 

Review the procedures for making these types of changes with respect to who is authorized to 
process the change and how difficult or easy it is to do. While this may not sound like a 
significant issue, these problems arise at the beginning of the semester when it is most hectic. 
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Reliance on the vendor or difficulty in making the change can delay students accessing the 
system and from doing the required work. 

• Screen view: When instructors respond to student questions, they generally have both the 
question and solution screen open simultaneously. Some systems open multiple windows 
allowing you to toggle back and forth or to move the windows on the screen. Some append the 
solution to the problem, which means you have to scroll across the screen to see the solution 
and then scroll back to see the problem. A review of the screen view for the product can help 
to ensure that they system meets your preferred method of navigation. 

• Providing student feedback: Some problems and exercises include short essay answers, 
which the instructor must grade. Some systems allow for electronic feedback for each problem 
and some do not. If they do not, then instructors must devise a way to give feedback and 
explanations of points awarded/deducted to students. 

• Print view: When working face to face with a student, it is convenient to be able to print out 
the student’s homework as well as the correct solutions. It is important to know the print 
options as it is impossible to print out the solution in some systems. 

• Ability to adjust student grades: All systems allow instructors to adjust grades. However, 
some systems only allow an adjustment if the student has attempted the assignment. In some 
cases, instructors may elect to accept a hard copy assignment (i.e., because the student was ill, 
because the student could not access the electronic system, etc.) In these cases, if the grade 
cannot be entered on the electronic system, the instructor must keep a separate record of that 
assignment until homework grades are downloaded at the end of the term. 

• Time out rules: As with any electronic system, there is a time out feature, resulting from non¬ 
use. Some systems warn users and some do not. Students tend to print out problems, work the 
solution, and then enter their answers. If they do not log out and back in, they may enter all 
their answers only to be told that they were “timed out” when they click “submit”. Knowing 
the time-out parameters can help alleviate user frustration and acceptance of the technology. 

Available Content 

Content may vary significantly across textbook publishers with respect to both the type and quantity of 
content and homework problems available. Central considerations relate to the breadth of electronic 
content and homework problems available. 

• Electronic technology content: The content available to students varies among products. The 
most comprehensive products include an electronic text, self-study questions and quizzes, and 
audio and power point lectures in addition to the homework. The least comprehensive include 
only the homework. Knowing what is included in the standard product will help when 
comparing alternative systems. If the product is only available with text plus ancillaries 
including web based homework, its cost and access code life will be significantly higher than a 
standalone homework system. 

• Quantity of problems available: The problems and exercises included in the electronic 
system are comprised of end-of-chapter and/or test bank problems and exercises. However, 
some systems include only selected homework items. It is important to understand what 
percentage of the possible problems is available electronically, and what types of problems are 
included. Some instructors do not want test bank questions included in the problem set. The 
answer will vary by textbook. 
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Relevant Features 

Web-based learning systems have unique features that may differ significantly across textbook 
publishers. The following represent key feature concerns (Pergola & Squires, 2007). 

• Algorithmic assignments of problems and solutions for instructors: One of the features of 
the systems allows for algorithmic creation of some problems and exercises, a feature that 
reduces cheating. This feature results in each student having a different set of values for each 
problem and in some cases, for each attempt. As students have questions about homework, the 
instructor must respond to their unique problem. An important consideration is whether the 
system allows instructors to view solutions to each problem attempt. Some systems do not 
allow instructors to view the solution until all attempts are used. This necessitates working the 
problem each time a student has a question. Ask if the system allows the instructor to view the 
solution for each attempt. The importance of this feature will differ depending upon type of 
course, material level, etc. Some instructors have indicated that they don’t care about 
algorithmic solution availability in principles courses because it’s easy to calculate the answer 
manually. In upper level classes where students have learned to appreciate the value of doing 
and learning from the homework, algorithmic problems may not be necessary. 

• Rounding issues: Student answers are sometimes marked incorrect because they have rounded 
their response differently than the answer key. Some systems allow an adjustment for rounding 
to avoid this. 

• Answer formats: There are a variety of vendor approaches to formatting. Some vendors 
define solution sets with all feasible formats (whole number, two decimal, currency formats, 
decimals, percentages, etc....); some vendors provide specific instructions that students must 
follow for an answer to be counted as correct; some vendors provide an answer field that is 
formatted in the desired format. It is important to understand how the system treats formatting 
to avoid student frustration and confusion that results in students questioning their 
understanding of the concept when their answer is, in fact, correct but formatted incorrectly. 

• Attempts parameters: The systems allow the instructor to define the number of attempts 
allowed for the assignment. However, some systems define attempts as the number allowed 
per problem and some define them as the number of attempts per problem set. In the latter 
case, instructors must decide how many attempts for each problem and then set the total 
attempts as a function of the number of problems and the number of attempts per problem (3 
attempts per problem, 4 problems, 12 total attempts). Students, however, can use all 12 
attempts on the first problem and have none left for the remainder of the assignment. 

• Grading parameters: Systems differ in recording of grades. Some systems record the final 
attempt grade and some record the highest grade of all attempts. There are also systems that 
allow the instructor to choose which grade is accepted. 

• Grading metrics: It is important to understand if the system supports your method of 
assignment points, i.e., points per problem, problem weighting, percentage assignment, etc. 

• Link-to-book option: One feature included in some systems is the ability to link to the book 
when answering questions. If a student submits a response that is incorrect, the link feature 
points them to the learning objective section for the question that they missed. 

• Ability to work ahead: Some systems allow students to work on subsequent assignments even 
if a previous assignment due date has not been reached. Some systems allow forward 
progression but block students from previous assignments once a new one has been started. 

• Solution (not just answer) provided to student: The systems differentiate between answers 
and solutions, where the solution shows how to derive the answer. Some systems provide the 
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solution to students after the final submission, if the instructor has chosen this as an allowed 
feature, and some do not. If no solutions are provided, instructors may have to post solutions 
elsewhere and/or work the problems in class. 

• Feedback prior to submission: Some systems provide a “check my work” or “how am I 
doing” feature that allows students to check their work prior to submission for grading. It is 
important to know if such a feature exists, if the instructor can control access to that feature, or 
if the instructor can control how that feature is accessed in the system. 

• Student access to homework after due date: Some systems allow students access after the 
due date as a function of the system, some require instructors to define accessibility, and some 
do not allow access at all. For systems that allow access after the due date, students can 
continue attempting the problem until they get it right without the attempts affecting their 
grade. The system will score the grade from the allowed attempts only. For systems in which 
instructors define access, they are a variety of options including no access, view only, and view 
with help. It is also important to know how late submissions are distinguished from those that 
are on time. 

• Hints: Some systems provide hints to the students as they work the problem. If the system has 
this feature, it is important to understand how it works. Consideration include if hints are 
predefined, if instructors can author hints, and if instructors can control how and when hints are 
used. 

SYSTEMS SELECTION 

Systems selection involves identifying feasible solutions and evaluating solutions based on the 
ability to satisfy system specifications (ASA Research 2010, Gel in as & Dull 2010, Gelinas, Sutton, & 
Hunton 2005). With respect to evaluating web-base learning systems, this would involve identifying 
the systems to be considered, evaluating system functions and features based on system specifications, 
and assessing system suitability based on system functions and features. 

Identifying Feasible Solutions 

As mentioned above most accounting textbooks now offer web-based learning systems as 
supplements to their accounting textbooks. Instructors must first identify the web-based learning 
system accompanying each textbook under consideration for adoption. This is easily accomplished by 
accessing publisher websites and reviewing the supplemental resources available for the textbooks of 
interest (e.g., see http://www.cengage.com/highered/ , http://www.pearsonhighered.com/educator , 
http://www.wiley.com/WileyCDA/ , and http://connect.mcgraw-hill.com/) . The most prominent web- 
based systems accompanying accounting textbooks are CengageNOW (Cengage-Thompson- 
Southwestern), MyAccountingLab (Pearson-Prentice Hall), WileyPLUS (Wiley), and Homework 
Manager/Connect (McGraw-Hill). While there are a number of web-based learning systems available, 
not all web-based learning systems are created equal. There are significant differences across 
publishers with respect each system’s interface, functions, content, features, and support. These 
differences may directly impact the effectiveness of a web-based learning system as an instructional 
tool and as a corollary, issues or problems related to system functionality, features, and/or support may 
impact the ultimate utility of the associated textbook as a pedagogical resource. Consequently, an 
evaluation of available web-based learning systems is an essential component of the textbook review 
process and should consist of a meticulous appraisal of system elements in light of instructor and 
student requirements and system specifications. 
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Systems Evaluation 

Systems evaluation involves assessing each potential system based its ability to satisfy systems 
specifications identified during the systems analysis phase (Gelinas & Dull 2010, Gelinas, Sutton, & 
Hunton 2005). Once potential web-based learning systems have been identified, instructors should 
schedule vendor demonstrations, survey instructors currently using the potential systems, and/or, if 
available, request trial instructor access as a basis for evaluating each system’s ability to satisfy user 
requirements and systems specifications. This can be a difficult and time consuming task given the 
functional intricacies, numerous features, and disparities in capabilities across available web-based 
learning systems, further complicated by the fact that the relative importance of system specifications 
will likely vary by instructor depending on pedagogical needs and preferences. To assist instructors in 
organizing and completing a systems evaluation for a web-based learning system, an assessment 
framework was developed based on the systems survey and systems analysis performed in the previous 
sections of this paper. The assessment framework is organized around the feasibility, functionality, 
content, and features specifications defined above and contains an assessment rubric explaining least 
desirable, adequate, and most desirable conditions for each specification consideration. The 
Framework for Evaluating Web-Based Learning Systems is presented in table form in the appendix. 

The framework maybe used as a reference tool by instructors during vendor interviews or 
demonstrations, peer interviews, beta testing, and/or trial testing. Instructors may also assign a point 
scale to the assessment rubric dimensions (e.g., 1 = least desirable, 2 = adequate, 3= most desirable) as 
a basis for facilitating relative assessments across different web-based learning systems. Additionally, 
instructors may want to adapt the point scale allowing greater or lesser emphasis for particular system 
specifications to represent relative importance of system specifications based on instructor preferences. 
The framework may also be used as a basis for comparing textbook adoption proposals. Finally, 
although this framework was developed for use in evaluation of web-based learning solutions 
accompanying accounting textbooks, it could be easily adapted to other functional areas that have an 
interest in evaluating similar web-based learning products. 

CLOSING REMARKS 

Today’s students require innovative teaching methodologies that appeal to their unique learning 
styles and technology expectations. Web-based learning systems provide a means to address the 
learning preferences of the current generation of accounting students and aid instructors in addressing 
these preferences. However, each system has unique features and differing capability levels that 
impact the ultimate utility and learning effectiveness of the system. This makes assessing web-based 
systems as part of textbook adoptions a critical and complicated task, particularly for instructors who 
have not used such systems in the past. 

The purpose of this paper was threefold. The paper was intended first, to explain the relevance 
of web-based learning systems for addressing the learning preferences of the current generation of 
accounting students and second, to emphasize the importance of carefully evaluating web-based 
learning systems as part of a textbook adoption decision. The final and central purpose of this paper 
was to offer accounting educators some practical guidance to assist them in evaluating potential web- 
based learning systems. It is hoped that this paper will encourage accounting educators to consider the 
importance of web-based learning systems as pedagogical tools and as such, motivate them to carefully 
evaluate each potential system as a central part of the textbook review process. 
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APPENDIX: FRAMEWORK FOR EVALUATING WEB-BASED LEARNING SYSTEMS 


SPECIFICATIONS 

LEAST 

DESIRABLE 

ADEQUATE 

MOST DESIRABLE 

Feasibility Considerations 

Pricing 

High 

Medium 

Low 

Option Pricing 

Price per option 

Package pricing 

All inclusive 

Access code life 

One term 

Two terms 

Longer-term 

Compatibility with other 
applications 

Is not compatible 

Allows exporting to 
Excel, Word 

Allows both importing 
and exporting 

Instructor training and 
support 

Training is in the 
form of manuals. 

Training includes a 
combination of 
manuals and web- 
based training. 

Training includes, 
manuals, webinars, 
peer support, and/or 
on-site training. 

Type of instructor support 

Telephone support 

Telephone support 
and/or email support 

Interactive online 
support. 

Hours of instructor support 

Support is 
available Monday 
through Friday, 9- 
5. 

Support is available 
Monday through 

Friday extended 
hours. 

Support is available 
seven days a week, 
extended hours. 

Response rates 

Response rates for 
each 

communication 
mode are not 
available and/or 
are too slow 

Response rates for 
each communication 
mode are acceptable 

Response rates for 
each communication 
mode are good. 

Instructor feedback 

There is no formal 
process for 
instructor - 
reported errors, 
suggestions, 
and/or desired 
features. 

There is a process for 
instructor reported 
errors, suggestions, 
and/or desired 
features but the 
vendor does not 
provide response 
statistics or feedback. 

There is a process for 
instructor reported 
errors, suggestions, 
and/or desired features 
that provide feedback 
to instructors. 

Student enrollment and 
system use instructions 

Vendor provides 
student 

instructions but 
they lack clarity. 

Vendor provides 
student instructions 
that are reasonably 
easy to follow. 

Vendor provides 
instructions that are 
reasonably easy to 
follow and live 
enrollment support. 

Student technical support 

Support is 
available Monday 
through Friday, 9- 
5. 

Support is available 
Monday through 

Friday extended hours 
that more closely fit 
student needs. 

Support is available 
seven days a week, 
extended hours. 

Type of student support 

Telephone support 

Telephone support 
and/or email support 

Interactive online 
support. 
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SPECIFICATIONS 

LEAST 

DESIRABLE 

ADEQUATE 

MOST DESIRABLE 

Vendor Reliability 

Capacity and 
performance data 
are not available 

Capacity and 
performance data 
show that the 
publisher network is 
overloaded. 

Capacity and 
performance data 
show that the 
publisher network is 
reliable and accessible 
to students when 
needed. 

System Functionality 

Course Admin Procedures 

Vendor controlled 

School administrator 

Instructor controlled 


Difficult 

Moderately difficult 

Easy 

Ease of creating 
assignments 

Difficult 

Moderately difficult 

Easy 

Ease of authoring 
problems/exercises 

Difficult 

Moderately difficult 

Easy 

Ability to support desired 
problem format 

Cannot 

Can to some degree 

Fully supports 

Ability to change 
assignment parameters 

Once created, 
cannot be changed 

Can be changed but 
may result in lost 
grades 

Can be changed and 
will not impact 
student attempts 

Student view 

Does not allow 
"student view" 

Allows "student 
view" but cannot 
"take assignments as 
a student" 

Allows student view 
and can take 
assignments as if you 
were the student 

Transfer of student access 
between classes 

Vendor controlled 

Can be done but is 
difficult 

Can be done and is 
easy to administer 

Screen view 

Solution is 
appended to the 
problem; must 
scroll back and 
forth to view 

System allows open 
windows of both the 
problem and solution; 
can toggle between 
them 

System allows open 
windows of problems 
and solutions open on 
the same screen 

Providing student feedback 

System does not 
allow student 
feedback 

System allows 
student feedback in 
comment form only 

System allows student 
feedback in multiple 
forms 

Print view 

Students cannot 
print the problems 

Students can print the 
problems but 
formatting cuts off 
part of the problem 

Students are provided 
with formatted 
printable problems 

Ability to adjust student 
grades 

Cannot adjust 
grades 

Cannot adjust grades 
if student has not 
attempted assignment 

Can adjust grades 
whether student has 
attempted the 
assignment or not. 
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SPECIFICATIONS 

LEAST 

DESIRABLE 

ADEQUATE 

MOST DESIRABLE 

Time out rules 

System has a time¬ 
out feature but 
does not warn the 
student. 

System has a time out 
feature and warns the 
student when they are 
"timed out". 

System has a time out 
feature and tells the 
student they have 
timed out if they 
attempt to enter data. 

Available Content 

Electronic Technology 
Content 

System includes 
only homework 
problems 

System includes 
homework and 
electronic text 

System includes 
homework, electronic 
text, and other 
features (audio 
lectures, power points, 
self-study etc...) 

Quantity of Problems 
Available 

The 

problems/exercises 
include only some 
of the end-of- 
chapter content 

The 

problems/exercises 
included contain all 
of the end of chapter 
content 

The 

problems/exercises 
included contain end- 
of-chapter content, 
test bank content, and 
self-study content 

Relevant Features 

Algorithmic problems and 
solutions 

Does not allow 

algorithmic 

problems 

Allows algorithmic 
problems but 
instructor cannot see 
the solution until 
after all attempts are 
used 

Allows algorithmic 
problems and 
instructor can see 
solution for each 
problem and attempt 

Rounding 

System does not 

accommodate 

rounding 

System 

accommodates 
rounding but the 
rounding parameter is 
too broad 

System allows for 
rounding within a 
"normal" rounding 
range 

Answer formats 

System is rigid, 
accepting only one 
format for an 

answer. 

System is somewhat 
flexible in answer 
formats. 

System anticipates 
most answer formats 
and accepts them as 
correct. 

Attempts Parameter 

Attempts 
definitions are too 
broad (unlimited) 
or too narrow (1 
attempt) 

System allows for 
user defined number 
of attempts but it is 
for the problem set, 
not per problem 

System allows for 
user defined number 
of attempts per 
problem, not problem 
set. 

Grade parameters 

System uses final 
attempt grade; 
instructor cannot 
define 

System uses the best 
of the attempts 

User can define which 
attempt should be 
used as the grade 
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SPECIFICATIONS 

LEAST 

DESIRABLE 

ADEQUATE 

MOST DESIRABLE 

Grading metrics 

System does not 
allow any (or very 
little) flexibility in 
grading metrics. 

System allows some 
flexibility regarding 
grading metrics but is 
limited in choice 

System allows user to 
define the grading 
metrics - points, 
percentages, weights 
etc... 

Link-to-book Option 

System does not 
include an e-book 
and/or link to e- 
book feature 

System includes an e- 
book but link to the 
content is too broad 

System includes an e- 
book and the link to 
the book is specific to 
the problem the 
student is having 

Ability to work ahead 

Students cannot 

access a new 
problem until the 
current problem or 
assignment is 
complete 

System allows 
forward progression 
but does not allow 
access to previous 
assignments once the 
new one is started. 

System allows 
forward progression 
with no restrictions. 

Solutions/Answers 

System provides 
the answer to the 
problem, but not 
the solution. 

System provides both 
the answer and the 
solution but not until 
after the due date. 

System provides both 
the answer and the 
solution after the final 
submission. 

Feedback prior to 
submission 

System provides a 
"check my work" 
or "how am I 
doing" feature that 
allows students to 
see if their answer 
is correct prior to 
submission. 

System provides a 
"check my work" or 
"how am I doing" 
feature that allows 
students to see if their 
answer is correct 
prior to submission 
but the feature is 
controlled by the 
instructor. 

System does not 
provide a "check my 
work" or "how am I 
doing feature". 

Student access to homework 
after due date 

System does not 
allow student 

access to 
homework after 
the due date. 

System does allow 
access to the 
homework as a 
function of the 
system. 

System does allow 
access according to 
instructor-defined 
accessibility 

Hints 

System does not 
provide hints. 

System provides pre¬ 
defined hints 

System provides pre¬ 
defined hints and also 
allows instructor- 
defined hints. 
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